Stabilization of captured images

ABSTRACT

Images captured by an image capture device on a line of sight with an associated gyrometric system designed to supply gyrometric measurement data relating to the line of sight are stabilized. To this end, a first series of captured images is obtained. Then, a second series of stabilized images is obtained by applying, to the first series of images, an image stabilization based on gyrometric information. Then, a residual stabilization error is determined by applying a digital processing to the second series of stabilized images. Finally, the preceding three steps are repeated. On initialization, the gyrometric information corresponds to the measurement data supplied by the gyrometric system. Then, subsequently, the gyrometric information is obtained by correcting, on the basis of the residual error, the measurement data supplied by the gyrometric system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under the Paris Convention to theFrench Patent Application No. 10 52371, filed on Mar. 30, 2010.

FIELD OF THE DISCLOSURE

The present invention relates to image processing and more particularlyto the stabilization of images captured by an image capture device, suchas a video camera for example.

When a camera is subjected to involuntary movements, such as, forexample, the movements of the holder of the device, the captured imagesmay be of poor quality. Indeed, the involuntary movements of the line ofsight of the camera may introduce a ‘movement blur’ which affects thequality of the captured images.

It is possible to envisage using data supplied by gyrometers as a basisfor stabilizing the line of sight and thus avoiding the movement blur.In this case, an image stabilization is applied on the basis of datasupplied by the gyrometers.

Alternatively, to digitally stabilize the captured images, notably byapplying a digital image processing such as that described in thedocument FR 2 828 315. In this case, the image stabilization is based onoffsets between the successive images which are determined by digitalprocessing.

Thus, the stabilization is either based on offsets supplied bygyrometers, or based on offsets determined by digital processing appliedto the images to be stabilized.

A document WO 2008/000711 describes a stabilization method which reliesboth on the use of data supplied by gyroscopes and on a digitalprocessing of the captured images. More specifically, according to theteaching of this document, firstly, approximate offsets between thesuccessive images are determined on the basis of gyrometricmeasurements, then fine offsets are determined on the basis of theapproximate offsets and the stream of images.

Then, provision is made to stabilize the images by applying either thefine offsets if they are considered as relevant, or the approximateoffsets in the case where the fine offsets obtained would not berelevant. To this end, there is therefore a step for assessing the fineoffsets in order to determine if it is relevant to apply them.

It should be noted that some known stabilization devices alreadyimplement a stabilization based on gyrometric measurements. In suchdevices, the implementation of a method such as that described in thedocument WO 2008/000711 requires a modification of the existingarchitecture. In practice, when a gyrometric stabilization isimplemented, the calculated offsets are applied to the captured images,and thus, at the output of this stabilization step, images stabilized onthe basis of the gyrometric measurements are obtained. Consequently, ifthere is a desire to assess the value of the fine offsets beforechoosing between fine offsets and approximate offsets, the existingarchitecture must be modified.

Furthermore, according to the teaching of document WO 2008/000711, insome cases, the image stabilization is performed only on the approximateoffsets. Now, these approximate offsets do not make it possible toobtain a fine and accurate stabilization. Thus, it is sometimes notpossible to obtain an efficient stabilization by applying the teachingof document WO 2008/000711.

SUMMARY

The present invention aims to improve the situation.

A first aspect of the present invention proposes a method forstabilizing images captured by an image capture device on a line ofsight with an associated gyrometric system designed to supply gyrometricmeasurement data relating to the line of sight,

said method comprising the following steps:

-   -   /a/ obtaining a first series of captured images;    -   /b/ obtaining a second series of stabilized images by applying,        to the first series of images, an image stabilization based on        gyrometric information;    -   /c/ determining a residual stabilization error by applying a        digital processing to the second series of stabilized images;        and    -   /d/ repeating steps /a/ to /c/;        wherein, on the first iteration of step /b/, the gyrometric        information corresponds to the measurement data supplied by the        gyrometric system, and on an iteration following step /b/, the        gyrometric information is obtained by correcting, on the basis        of said residual error, the data supplied by the gyrometric        system.

The expression ‘image stabilization’ should be understood to mean anoperation consisting in realigning the successive images relative to oneanother, in order to reduce or even cancel out the effects of movementsof the line of sight during the capture of these images. In practice,between two successive images, the line of sight may have undergoneroll, pitch or yaw rotations or even changes of focal length, in thecase of a camera for which the zoom factor can be varied, translationalmovements and angular and linear vibrations. The image stabilizationtherefore aims to cancel out these movements which affect the quality ofthe captured images, by compensating them.

Here, provision is made on the one hand to use gyrometric measurementdata supplied by the gyrometric system to stabilize the captured images,and, on the other hand, to use a digital processing to correct thesegyrometric measurement data. Thus, in these conditions, it is possibleto stabilize the captured images accurately on the basis of thecorrected gyroscopic measurement data. The accuracy and the reliabilityof such a stabilization makes it possible both to obtain a second seriesof quality images and to reduce the complexity of the calculationsperformed while the digital processing is being implemented in the step/c/. In practice, since the second series of images exhibits a reliablestabilization, the residual error obtained by applying the digitalprocessing is easy to determine.

It should be noted that each gyrometer has characteristic parameters.Obtaining these characteristic parameters accurately and preciselycurrently entails costly measurement operations in the manufacturingphase. Furthermore, these characteristic parameters which are measuredduring manufacture may not be stable over time. Such is the case inparticular for inexpensive gyrometers. Thus, advantageously, by virtueof the retroactive correction of the gyrometric information, it is herepossible to stabilize the captured images accurately even when thegyrometric system consists of inexpensive gyrometers.

It is easy for those skilled in the art to determine offsets between thecaptured images in order to stabilize them on the basis of rawgyrometric measurement data supplied by the gyrometers used. To thisend, provision may be made to use a mathematical model which takes intoaccount a greater or lesser number of characteristic parameters of thegyrometric system used and of its operating environment (temperature,accelerations, statistical or frequential model of the involuntary anddeliberate movements, etc.). The accuracy and the number of thecharacteristic parameters taken into account in the mathematical modelmake it possible to improve the accuracy of the offsets obtained. Then,on the basis of such a model, it is possible to correct, in a relevantand accurate manner, the gyroscopic measurement data supplied by thegyroscopic system, even when these data are supplied by inaccurategyrometers.

Furthermore, it is easily possible to implement an image stabilizationin the context of an already existing stabilization architecture, whichis based on gyrometric measurement data. In practice, it is simple toadd, in series, without having to modify the existing architecturerelating to a stabilization relying on gyrometric measurement data, astep of applying digital processing that aims to determine a residualstabilization error. It is possible here to obtain, on the basis of thegyrometric measurement data, an offset between two captured images withan accuracy of the order of 1 pixel, when the data supplied by thegyrometric system are corrected retroactively according to oneembodiment of the present invention.

By virtue of these retroactive corrections, the gyrometric informationmakes it possible to accurately calculate offsets between the capturedsuccessive images, and no longer only approximate offsets, as is thecase with the prior art.

More specifically, the gyrometric information indicates the movements ofthe line of sight and therefore makes it possible to realign thecaptured images relative to one another by translating the movements ofthe line of sight in the form of pixel displacements in the capturedimages.

The stabilization based on the gyrometric measurement data is a reliablestabilization by virtue of the retroactive correction of thesegyrometric measurement data. This characteristic makes it possible toobtain a series of stabilized captured images with high qualityregardless of the type of the scenes captured, including when a digitalprocessing is not suitable for calculating relevant and reliableoffsets. Such may be the case, for example when the images do notcontain enough information, such as an image of substantially uniformcolour, or when the image is crossed by a moving object moving at highspeed disturbing the calculation of the offsets by digital processing.In these cases, only the approximate offsets may be applied in the priorart, whereas here, the stabilization obtained is accurate.

Indeed, in one embodiment of the present invention, by virtue of thisretroactive correction of the gyrometric system, the stabilization basedon the movements of the line of sight makes it possible to obtainrelevant results even in complicated cases, and even if the gyrometersused are not of high quality.

Provision may be made for the stabilization method to also include,after step /c/, the following step:

-   -   obtaining a third series of stabilized images by applying, to        the second series of stabilized images, a second image        stabilization based on a digital image processing taking into        account the residual error determined at the preceding step /c/.

In this embodiment, advantageously, the third series of images obtainedultimately may exhibit a stabilization with very high accuracy.

The stabilization which is based on a digital processing relies on arelative analysis of the pixels of the successive images. There is nolimitation on such a digital stabilization processing. Provision may bemade to determine offsets between two successive images on the basis ofa digital correlation such as that described in the document FR 2 828315, based on an optical stream method.

There is no limitation on the gyrometric system with respect to thepresent invention. Such a system may comprise a number of gyroscopes. Itis possible, for example, to provide for the use of two gyrometersrelative to a benchmark image plane and a third gyrometer to determinethe information concerning the rotation of the image about the line ofsight. It is designed to supply gyrometric measurement data relating tothe movement of the line of sight of the capture device. To this end,provision may be made for the gyrometric system to be securely attachedto the line of sight of the capture device.

More specifically, at step /b/, first offsets between the successivecaptured images can be determined on the basis of the movements of theline of sight indicated by the gyrometric information; and

at step /c/, second offsets between successive images of the secondseries can be determined on the basis of the digital image processing.

-   -   Thus, first offsets are determined on the basis of gyrometric        information and second offsets are determined on the basis of a        digital processing for a possible second complementary        stabilization, which may advantageously make it possible to        ultimately obtain a very efficient result.

It should be noted here that the first offsets are calculated andapplied first, and independently of the second offsets, unlike in theprior art. This independence of calculation advantageously makes itpossible to easily implement an embodiment of the present invention inan existing architecture.

In one embodiment of the present invention, applying a filtering duringthe stabilization process is envisaged. The stabilization at step /b/ isthen based on gyrometric information filtered by a low-pass filter.

The aim of such filtering is advantageously to filter out theinvoluntary movements from all the detected voluntary and involuntarymovements of the line of sight so as to make it possible to realign theimages relative to these involuntary movements. The voluntary movementsare generally characterized by a high amplitude and a low frequency. Alow-pass filter is suitable for this purpose.

In one embodiment, provision is also made to use the residualstabilization errors as a basis for modelling certain corrections of thegyrometers according to external parameters. Thus, it is advantageouslypossible to model a correction of the gyrometers according totemperature variations.

Indeed, when a temperature sensor can supply temperature values, thevariations of the residual stabilization errors according to temperaturevariations can be recorded.

This modelling is constructed during the implementation of the methodand remains easier to perform than a correction modelling conventionallyperformed when constructing the gyrometric system in the factory.Furthermore, such a correction modelling according to one embodiment ofthe present invention may be suited to the modifications that thegyrometers undergo over time since it can easily be updated regularly.

This type of correction model can advantageously be used subsequently onthe simple basis of the knowledge of the temperature and no longer onthe basis of the residual stabilization error.

Thus, in one embodiment of the present invention, there is also obtaineda model for correcting, according to temperatures, gyrometricmeasurement data on the basis of the residual stabilization errorsobtained.

Provision can notably be made to correct the gyroscopic measurement dataon such a correction model at the start of the series of the capturedimages, or else when the captured images do not contain enoughinformation.

In one embodiment of the present invention, step /b/ is based ongyrometric measurement data filtered on the basis of the characteristicsof the environment of the gyrometric system.

Provision is made to determine a correction value relative to thegyrometric system according to the residual error obtained at step /c/.This correction value can be obtained by applying a digital filter. Thisfilter uses the residual stabilization errors measured by the digitalimage processing to automatically refresh the characteristic parametersof the gyrometric system used to correct the measurement data supplieddirectly by the gyrometric system.

A second aspect of the present invention proposes a stabilization devicecomprising means designed to implement a stabilization method accordingto the first aspect of the present invention.

A third aspect of the present invention proposes an image capture systemcomprising an image capture device designed to capture a series ofimages on a line of sight, a gyrometric system designed to supplygyrometric measurement data relating to the line of sight and an imagestabilization device according to the second aspect of the presentinvention.

Other aspects, aims and advantages of the invention will become apparentfrom reading the description of one of its embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will also be better understood with the help of thedrawings, in which:

FIG. 1 illustrates the main steps of a method according to oneembodiment of the present invention;

FIG. 2 illustrates detailed steps of a method according to oneembodiment of the present invention;

FIG. 3 illustrates the image stabilization device according to oneembodiment of the present invention; and

FIG. 4 illustrates a filtering according to one embodiment of thepresent invention.

DETAILED DESCRIPTION

In one embodiment of the present invention, a stabilization method isimplemented in a stabilization device which is designed to receive aseries of images captured by an image capture device, such as a camera,on a line of sight with an associated gyrometric system. The latter isassociated with the line of sight of the image capture device so thatthe measurement data supplied by the gyrometric system indicatemovements of this line of sight.

FIG. 1 illustrates the main steps of a method according to oneembodiment of the present invention.

While this method is being applied, images are successively captured bythe image capture device and a first series of captured images istherefore obtained at a step 11.

The gyrometric system supplies gyrometric measurement data which relateto the movements of the line of sight of the capture device. Thesemeasurement data therefore make it possible to determine the movementsthat the line of sight of the capture device undergoes. However, thesemeasurement data may be more or less accurate.

At a step 12, a stabilization is applied to the series of capturedimages, on the basis of gyrometric information. The stabilization methodis an iterative method. At the first step, the stabilization isperformed on the basis of the gyrometric measurement data directlysupplied by the gyrometric system. Thus, on initialization, thegyroscopic information corresponds to the raw measurement data suppliedby the system. Then, for the subsequent iteration or iterations, thisgyrometric information is advantageously derived from the measurementdata corrected on the basis of a residual error as defined hereinbelow.

At this step 12, the gyrometric information is taken into account todetermine first offsets between the successive captured images. Thecaptured images can then be realigned relative to one another on thebasis of the movement of the line of sight indicated by the gyrometricinformation. On completion of this first stabilization, a second seriesof images is obtained.

At a step 13, a residual stabilization error is determined relative tothis second series of images by applying a digital processing. Such adigital processing may be that which is described in the document FR 2828 315, based on an optical stream method.

Thus, firstly, at step 12, provision is made to determine the offsets tobe corrected to stabilize the captured images on the basis of gyrometricinformation and, secondly, at step 13, provision is made to determineresidual offset errors still remaining in the second series ofstabilized images, on the basis of a digital processing.

The residual errors 13 are transmitted to the gyrometric information, sothat the stabilization 12 is advantageously performed by taking intoaccount the corrected gyrometric information.

By virtue of this step 13 for determining the residual error, the datasupplied by the gyrometric system can be retroactively corrected.Depending on the context, notably according to the desired stabilizationaccuracy or else according to temperature variations undergone by thegyrometric system, provision can be made to correct the data from thegyrometric system more or less often.

FIG. 2 illustrates a stabilization method in one embodiment of thepresent invention. In this method, the part relating to the gyrometricstabilization 12 is detailed. Thus, the stabilization 11 here comprisesa correction substep 23 responsible for retroactively correcting themeasurement data 22 supplied by the gyrometric system. To this end, boththe data supplied by the gyrometric system and characteristic parametersof this system which are advantageously updated, in a step 28, on thebasis of step 13, are taken into account.

A substep for determining offsets 24 between successive images isresponsible for receiving, on the one hand, the successive capturedimages and on the other hand gyrometric information obtained bycorrection of the measurement data. Then, a stabilization substep 25 isresponsible for applying the offsets determined at step 24 to the imagescaptured at step 11.

As indicated above, this method comprises a substep for management ofthe characteristic parameters 28 of the gyrometers of the gyrometricsystem considered here. This management substep 28 is responsible forupdating the characteristic parameters of the gyrometers of the systemon the basis of the residual stabilization errors determined over timeat step 13.

Furthermore, provision can be made here to apply a digital stabilizationto the second series of images obtained on completion of step 25. Inpractice, it is advantageously possible here to use the residual errorcalculated in the preceding step 13 to apply a second stabilization 27to the second series of images in order to obtain a third series ofstabilized images. This third series of images may advantageouslyexhibit a high quality level.

It can clearly be seen that the method according to one embodiment ofthe present invention can be easily implemented in an already existingarchitecture within which an image offset correction is already appliedbased on gyrometric measurement data.

FIG. 3 illustrates an architecture of an image capture system accordingto one embodiment of the present invention comprising an imagestabilization device according to one embodiment of the presentinvention, cooperating with an image capture device and a gyrometricsystem.

Such a captured image stabilization device 300 is designed to stabilizethe images captured by an image capture device 32 on a line of sight 33with an associated gyrometric system 22.

This stabilization system comprises:

said stabilization device comprising:

-   -   a first interface unit 303 designed to receive a first series of        captured images 201;    -   a second interface unit 304 designed to receive gyrometric        measurement data from the gyrometric system;    -   a stabilization unit 305 designed:        -   to apply an image stabilization based on gyrometric            information; and        -   to supply a second series of images 202;    -   a determination unit 307 designed to determine a residual        stabilization error by applying a digital processing to the        second series of stabilized images;    -   a measurement correction unit 306 designed to supply the        gyrometric information by correcting the measurement data        supplied for the gyrometric system on the basis of the residual        stabilization error.

Depending on the context of use of an image capture system, there may bea desire to stabilize the captured images relative to all the movementsof the line of sight. In this case, the first stabilization may directlytake into account the gyroscopic information as described hereinabove.

In other contexts of use, it may be preferable to stabilize the capturedimages only relative to involuntary movements of the line of sight, butnot relative to the voluntary movements. To this end, it is possible toprovide for the gyrometric information making it possible to determinethe movements of the line of sight of the capture device to be filtered.

The second interface unit 304 may then also comprise a low-pass filterdesigned to filter the gyrometric information. This low-pass filter isdesigned to filter out the voluntary movements and thus allow for arealignment (or stabilization) of the captured images on the basis ofthe involuntary movements, as is described hereinbelow with reference toFIG. 4.

It is also possible to provide for another filtering to be applied tothe gyrometric measurement data supplied by the gyrometric system. Thisfiltering corresponds to the determination of gyroscopic measurementdata correction values and it is implemented in the measurementcorrection unit 306. This filter may take into account characteristicparameters of the gyrometers used which may be obtained from residualerrors relating to the gyrometric system determined while implementing amethod according to one embodiment of the present invention.

FIG. 4 illustrates the application of the filtering to the movements.The movements detected by the gyrometric system may correspond both tovoluntary movements of the line of sight and to involuntary movements.The detected movements 41 are illustrated on a curve representing thefrequency of the movements on the X-axis and their amplitude on theY-axis. The movements that have a frequency greater than a maximumthreshold frequency F_(max) are filtered because they are considered asinvoluntary movements.

This filtering makes it possible to stabilize the captured images 201relative to the involuntary movements of the line of sight.

Advantageously, the retroactive correction of the gyrometric systemmakes it possible to avoid drifts over time in the gyrometricmeasurements and thereby obtain an accurate stabilization regardless ofthe conditions of use of the image capture system.

Furthermore, given that the gyrometric information is accurate andreliable, the architecture of the stabilization device according to oneembodiment of the present invention makes it possible to substantiallyreduce the complexity of the digital calculations performed.

1. A method for stabilizing images captured by an image capture device on a line of sight with an associated gyrometric system designed to supply gyrometric measurement data relating to the line of sight, said method comprising the following steps: /a/ obtaining a first series of captured images; /b/ obtaining a second series of stabilized images by applying, to the first series of images, an image stabilization based on gyrometric information; /c/ determining a residual stabilization error by applying a digital processing to the second series of stabilized images; and /d/ repeating steps /a/ to /c/; wherein, on the first iteration of step /b/, the gyrometric information corresponds to the measurement data supplied by the gyrometric system, and on an iteration following step /b/, the gyrometric information is obtained by correcting, on the basis of said residual error, the measurement data supplied by the gyrometric system.
 2. The image stabilization method according to claim 1, also comprising, after step /c/, the following step: obtaining a third series of stabilized images by applying, to the second series of stabilized images, a second image stabilization based on a digital image processing taking into account the residual error determined at the preceding step /c/.
 3. The image stabilization method according to claim 2, wherein, at step /b/, first offsets between the successive captured images are determined on the basis of the movements of the line of sight indicated by the gyrometric information; and wherein, at step /c/, second offsets between successive images of the second series are determined on the basis of the digital image processing.
 4. The image stabilization method according to claim 1, wherein the gyrometric information is filtered by a low-pass filter.
 5. The image stabilization method according to claim 1, wherein there is also obtained a model for correcting, according to temperatures, gyrometric measurement data on the basis of the residual stabilization errors obtained.
 6. A device for stabilizing images captured by an image capture device on a line of sight with an associated gyrometric system designed to supply gyrometric measurement data relating to the line of sight, said stabilization device comprising: a first interface unit designed to receive a first series of captured images; a second interface unit designed to receive gyrometric measurement data from the gyrometric system; a stabilization unit designed: to apply to the first series of captured images an image stabilization based on gyrometric information in order to obtain a second series of images; and to supply the second series of images; a determination unit designed to determine a residual stabilization error by applying a digital processing to the second series of stabilized images; a measurement correction unit designed to supply the gyrometric information by correcting the measurement data supplied for the gyrometric system on the basis of the residual stabilization error.
 7. The image stabilization device according to claim 6, wherein the second interface unit also comprises a low-pass filter designed to filter the gyrometric information.
 8. An image capture system comprising an image capture device designed to capture a series of images on a line of sight, a gyrometric system designed to supply gyrometric measurement data relating to the line of sight and an image stabilization device comprising: a first interface unit designed to receive the first series of captured images; a second interface unit designed to receive gyrometric measurement data from the gyrometric system; a stabilization unit designed: to apply to the first series of captured images an image stabilization based on gyrometric information in order to obtain a second series of images; and to supply the second series of images; a determination unit designed to determine a residual stabilization error by applying a digital processing to the second series of stabilized images; a measurement correction unit designed to supply the gyrometric information by correcting the measurement data supplied for the gyrometric system on the basis of the residual stabilization error.
 9. The image capture system according to claim 8, wherein the second interface unit also comprises a low-pass filter designed to filter the gyrometric information. 